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I .   INTRODUCTION 

A.   GENERAL 

The  goal  of  operational  testing  is  to  "conduct  field 
testing,  under  realistic  combat  conditions,  of  any  item  of 
weapons,  equipment,  or  munitions  for  the  purpose  of 
determining  the  effectiveness  and  suitability  for  use  in 
combat  by  typical  military  users"  [Ref.  1].  Unfortunately, 
due  to  budget  constraints  and  the  lack  of  maneuver  area, 
equipment,  troops  and  time,  it  may  not  be  feasible  to  test 
these  items  thoroughly.  The  use  of  modeling  and  simulation 
can  help  to  close  the  gap  when  full  scale  testing  is 
unavailable  [Ref.  2:p.  1].  Janus (A)  is  one  of  the  models  used 
for  this  purpose. 

I  Comparison  between  operational  field  tests  and  high 
resolution  combat  models  such  as  Janus  can  provide  more 
information  to  assist  test  design  and  possibly  extend  test 
procedures.  Once  a  field  test  has  been  performed,  the  data 
obtained  can  be  used  to  calibrate  the  model  to  the  test. 
Calibration  entails  making  adjustments  to  input  data  or  model 
logic  to  obtain  a  closer  agreement  with  an  external  index 
[Ref.  3:p.5].  If  the  model  proves  to  be  credible,  it  can  be 
used  to  extend  test  results  beyond  the  field  test  environment, 
thus  saving  time  and  money  which  would  be  required  to  run 


further  field  tests.  The  Army  uses  a  concept  called  Model- 
Test-Model  (M-T-M)  to  implement  modeling  and  simulation  in 
operational  testing. 

B.  THESIS  OBJECTIVE 

This  thesis  analyzes  the  feasibility  of  accrediting  the 
Janus  combat  model  for  the  post-test  modeling  of  aircraft 
engagement  ranges.  Specifically,  helicopter  engagement  ranges 
collected  from  the  Line-of -Sight  Forward  (Heavy)  (LOS-F-H) 
Initial  Operational  Field  Test  and  Evaluation  (IOTE)  conducted 
at  Fort  Hunter  Liggett,  California  in  Spring  1990  are  compared 
to  similar  ranges  generated  by  simulation  of  the  test  in  the 
Janus  combat  model.  Route  data  collected  from  the  operational 
test  are  replicated  within  Janus  so  that  runs  can  be  made  to 
obtain  model  engagement  ranges.  The  means  and  distributions 
of  engagement  ranges  from  both  the  model  and  operational  test 
are  compared  statistically  to  analyze  the  feasibility  of  using 
the  simulation  for  future  post-test  modeling  of  the  LOS-F-H 
system. 

C.  MODEL  ACCREDITATION 

Model  accreditation  is  part  of  the  process  of  validating 
or  establishing  credibility  of  the  model.  Validation  is  the 
"process  of  determining  that  a  model  is  an  accurate 
representation  of  the  intended  real-world  entity  from  the 
perspective  of  the  intended  use  of  the  model"  [Ref.  3:p.  1: 


Enclosure  2].  Accreditation  is  necessary  if  the  model  is  not 
fully  validated.  It  is  the  process  of  certifying  that  a  model 
achieved  an  established  standard  when  applied  for  a  specific 
purpose  [Ref.  2:p.  6].  Models  are  accredited  for  particular 
types  of  applications  since  validation  is  a  continuous  process 
and  full  validation  may  not  be  technically  or  economically 
feasible.  A  model  is  subject  to  accreditation  when  it  is 
proposed  for  use  with  a  new  application  or  system  [Ref.  3:p. 
3].  In  this  case,  the  Janus  combat  model  requires 
accreditation  with  the  new  LOS-F-H  system  performing 
helicopter  engagements. 

D.   DEFINITION  OF  ENGAGEMENT 

An  engagement  is  defined  as  the  moment  the  fire  button  is 
pushed  on  the  LOS-F-H  system.  Of  primary  interest  is  the 
range  between  the  target  and  the  system  when  engagement 
occurs.  This  analysis  will  be  concerned  only  with  the  range 
of  first  engagement.  This  is  the  initial  range  at  which  the 
gunner  shoots  at  a  target  for  the  first  time.  Consecutive 
shots  by  the  same  system  on  the  same  target  will  not  be 
considered.  These  consecutive  shots  are  assumed  to  be 
statistically  dependent  for  the  purpose  of  this  thesis.  Both 
Janus  and  the  LOS-F-H  data  will  be  edited  to  provide  the  range 
of  first  engagement. 


E.   LINE-OF-SIGHT  FORWARD  (HEAVY)  SYSTEM  [Ref  4:p.  1-7] 

Recently,  the  Army  has  proposed  a  major  revision  in 
divisional  air  defense  concept.  The  Forward  Area  Air  Defense 
System  ( FAADS )  program  integrates  five  components  designed  to 
protect  a  division  from  low  altitude  air  threat.  One  of  the 
primary  components  in  a  heavy  division  is  the  LOS-F-H  air 
defense  system.  The  LOS-F-H  is  a  surface  to  air  missile 
system  mounted  on  a  modified  Bradley  fighting  vehicle,  manned 
by  a  crew  of  three:  a  commander /radar  operator  (RO),  a 
gunner/electro-optics  ( EO )  operator,  and  a  driver  (Figure  1). 


M3A1  BRADLEY  CHASSIS 


EO  OPERATOR 
(GUNNER) 


RADAR  OPERATOR 
(SQUAD    LEADER) 


Figure    1    LOS-F-H    System 


The  fire  unit  has  its  own  acquisition  and  tracking  radar 
which  is  capable  of  tracking  several  aircraft  while  scanning 
for  others  out  to  20  kilometers.  The  system  carries  eight 
missiles  which  weigh  112  pounds  each  and  are  in  ready-to-fire 
canisters.  Each  missile  is  a  laser  beam  rider  guided  by  a 
coded  laser  beam  which  minimizes  the  effects  of 
countermeasure.  The  unclassified  range  of  the  missile  is  8 
kilometers . 

The  crew  uses  a  Forward  Looking  Infrared  sensor  (FLIR)  or 
an  optical  (TV)  sensor  to  help  identify  and  track  the  target. 
An  Identification,  Friend  or  Foe  (IFF)  device  which  receives 
an  encrypted  transmission  sent  by  friendly  aircraft  is  used  to 
make  final  identification  of  an  enemy  aircraft.  The  gunner 
then  launches  a  missile  and  tracks  it  to  the  target. 

F.   MODEL-TEST-MODEL  CONCEPT 

There  are  three  phases  to  the  Model-Test-Model  (M-T-M) 
concept:  pretest  modeling  phase,  field  test  and  modeling 
phase,  post-test  modeling  phase  [Ref.  5:p.3]. 

1.   Pretest  Phase 

This  phase  is  designed  to  assist  planners  prior  to 
actual  operational  testing.  Selected  models  exercise  possible 
test  scenarios  to  provide  information  on  the  best  possible 
test  design.  This  information  includes  methods  to  optimize 
data  collection,  minimize  test  failure  and  alert  testers  to 
the  impact  of  external  constraints.   The  results  provide 


planners  confidence  that  test  objectives  can  be  achieved  and 
provide  the  modeler  important  information  that  can  be  used 
with  future  modeling  of  the  system.  This  phase  can  save  the 
designer  valuable  time  which  would  otherwise  be  wasted  in  the 
field  [Ref  6],  Unfortunately,  this  phase  was  not  conducted 
prior  to  the  field  test  for  the  LOS-F-H. 
2.   Field  Test  and  Modeling  Phase 

This  phase  involves  a  series  of  complete  field  tests 
to  evaluate  the  new  system  and  model  runs  to  provide  rapid 
analysis  and  feedback  to  test  personnel.  Operational  tests 
are  run  in  order  to  provide  an  assessment  of  how  the  system 
characteristics  perform  in  a  variety  of  different  operational 
scenarios.  Under  the  best  circumstances,  scenarios  are 
developed  in  the  pre-test  phase  for  use  in  the  initial 
operational  trials.  Field  testing  is  done  by  military 
personnel  in  a  series  of  trials  conducted  to  replicate  the  use 
of  the  system.  These  trials  are  usually  between  two  opposing 
forces.  Once  a  field  test  is  complete  the  data  is  passed  to 
the  modeler  who  performs  successive  iterations  of  model  runs 
to  provide  feedback,  but  most  importantly  to  calibrate  the 
model  code  and  model  data  [Ref  5].  This  involves  updating 
input  parameters  such  as  weapon  characteristics  to  field 
constraints.  Actual  play  positions  are  used  as  input  to 
complete  the  model  runs  and  a  comparison  analysis  is  conducted 


at  the  individual  event  level  (i.e.  engagements)  to  determine 
the  possibility  of  accreditation. 
3.   Post-Test  Modeling 

The  third  and  final  phase  involves  the  cautious  use  of 
the  'calibrated'  model  in  order  to  extend  the  test  results  to 
conditions,  situations  and  threats  not  tested  in  the  field 
[Ref  5].  This  'non-testing'  may  be  due  to  cost,  safety, 
environmental,  equipment  or  some  other  type  of  constraint. 
During  this  phase  it  is  important  not  to  extend  the  model 
beyond  the  point  at  which  the  calibrated  process 
representations  would  no  longer  be  valid. 


II.   OPERATIONAL  FIELD  TEST 

A.  BACKGROUND 

The  operational  field  test  of  the  LOS-F-H  system  consisted 
of  fifty  maneuver  trials.  These  trials  were  conducted  from  9 
April  to  23  May  1990.  Each  trial  was  a  force  on  force  battle 
which  generally  lasted  one  hour.  Both  day  and  night  trials 
were  performed,  as  well  as  trials  in  MOPP0  and  MOPP4  (two 
variations  of  Mission  Oriented  Protective  Posture).  The 
battles  involved  Red  and  Blue  mechanized  forces  of  battalion 
(minus)  strength.  Battalion  (minus)  indicates  that  only  part 
of  the  battalion  was  used.  Surrogate  aircraft  representing 
the  Mi-24  Attack  Helicopter  (HIND),  the  Mi-14  Medium 
Helicopter  (HAVOC)  and  the  Mi-8  Medium  Helicopter  (HIP) 
supported  the  Red  forces.  The  LOS-F-H  mission  was  to  defend 
the  Blue  maneuver  force  against  air  attack  as  the  force 
conducted  its  mission.  Different  scenarios  were  tested  with 
variations  in  offensive  and  defensive  operations  and  Blue 
force  orientation. 

B.  AIRCRAFT  PRESENTATION 

Tactical  and  safety  controls  were  provided  for  all 
aircraft.  Airspace  safety  control  was  of  primary  concern  and 
sometimes  dictated  flight  routes,  altitudes  and  tactics.  The 
maximum  altitude  above  the  terrain  for  all  helicopters  was  150 


feet,  but  there  was  no  minimum  altitude.  Helicopters  entered 
the  battle  area  making  their  approach  to  the  Blue  player's 
front.  It  was  assumed  that  Blue  players  received  lateral 
protection  from  notional  units  which  are  units  that  are  not 
physically  on  the  ground,  but  are  perceived  to  be  there  by  all 
players.  Aircraft  were  accounted  for  in  presentations.  A 
presentation  began  when  an  aircraft  left  its  holding  area  to 
enter  battle  and  ended  when  it  returned  to  its  holding  area. 
If  an  aircraft  was  'killed'  during  a  presentation,  it  would 
return  to  its  holding  area  and  would  be  revived  by  the 
controllers  and  sent  into  battle  as  another  aircraft. 
Aircraft  were  the  only  players  revived  during  the  trial  and 
that  occurred  only  in  holding  areas  [Ref.  4:p.2-22]. 
Presentations  lasted  anywhere  from  7  to  30  minutes  depending 
on  the  trial  and  number  of  aircraft  desired  to  represent. 

C.   DATA  COLLECTION 

Firing  information  for  all  weapon  types  were  provided  by 
a  laser  installed  on  each  firer  and  by  laser  sensors  on  each 
target.  A  laser  pairing  provided  the  real-time  computer  with 
the  firer  and  target  identification.  This  process  was  called 
Real-time  Casualty  Assessment  (RTCA).  The  RTCA  consisted  of 
firer  and  target  identification,  weapon  type,  trigger  pull 
time,  probability  of  kill,  assessment  of  shot  and  time  of 
impact.  Video  tapes  were  used  inside  the  LOS-F-H  fire  unit  to 
record  the  battle  and  a  Range  Measuring  System  (RMS)  recorded 


the   system   location   data   as   well   as   engagement   range 
information  using  a  form  of  triangulation  [Ref  4]. 

Several  computer  reports  were  generated  providing 
information  for  analysis.  Two  of  these  files,  the  Player 
Position  Location  File  (PLS)  and  the  Attack  Engagement  File 
(AEF)  provided  information  specifically  concerning  the 
LOS-F-H.  The  PLS  file  recorded  player  locations  at  every 
second  during  the  battle.  This  data  was  used  to  imitate  the 
actual  battle  in  Janus.  The  AEF  file  provided  information  on 
the  trial  engagement  segment,  specifically  the  LOS-F-H 
engagement  of  aircraft.  The  data  from  this  file  were  used  for 
engagement  range  comparison  with  the  data  obtained  from  the 
Janus  runs. 

D.   DATA  LIMITATIONS 

1.   Range  Measuring  System  (RMS)  Errors 

This  type  of  error  affected  the  vehicle  and  aircraft 
routes  as  well  as  engagement  ranges.  The  RMS  at  Fort  Hunter 
Liggett  records  the  position  data  and  calculates  the 
engagement  range  from  the  position  data.  Errors  due  to 
inaccurate  triangulation  (spikes)  or  lost  signals  (gaps)  could 
provide  false  location  and  engagement  ranges.  Although  most 
of  the  data  was  smoothed  there  was  still  some  error  associated 
with  this  problem.  This  error  can  affect  the  actual  player 
location  as  well  as  the  calculated  engagement  range. 
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2.   Real-Time  Casualty  Assessment  (RTCA)  Errors 

RTCA  missed  about  18  percent  of  the  LOS-F-H  launches 
because  of  low  or  dead  batteries  in  the  laser  equipment. 
Other  launches  resulted  in  no  laser  pairing  and  therefore 
provided  no  target.  These  launches  were  reconstructed  as 
closely  as  possible  by  analyzing  video  tape  and  using  player 
location  plots.  Over  90  percent  of  the  total  launches  were 
evaluated  by  analyzing  video  and  verifying  RTCA  results 
leaving  almost  ten  percent  of  the  RTCA  results  not  verified. 

E.   DATA  SELECTION 

Although  the  test  consisted  of  fifty  trials,  only  five 
were  selected  for  analysis.  The  five  trials  chosen  were  under 
identical  conditions  of  daytime,  MOPP0,  no  smoke,  Blue  in 
defense  and  facing  northwest.  These  trials  were  selected 
since  they  provided  the  most  helicopter  engagement  data 
(Table  1) . 

TABLE  1  TRIAL  CONDITIONS 


LOS-F-H  TRIALS  SELECTED: 
100B,  112B,  122B,  123B,  125B 

FACTOR 

CONDITION 

TIME 

DAY 

TACTICS 

DEFENSE 

DIRECTION 

NORTHWEST 

SMOKE 

NONE 

CHEMICAL 

NONE 
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III.   JANUS 

A.   BACKGROUND 

Janus  is  an  interactive,  brigade  level,  two  sided,  event 
driven  simulation  that  models  fighting  systems  as  entities 
(helicopter,  tanks,  etc.).  Entity  characteristics  include 
descriptions  of  weapons  carried,  weapon  capabilities,  movement 
speeds  and  how  they  are  affected  by  terrain,  ammunition  and 
fuel,  crew  performance,  sensor  data  and  supply /resupply  data. 
The  data  can  be  interactively  reviewed  and  changed  by  the 
user.  As  with  any  simulation  an  accurate  and  complete 
database  is  crucial  to  operation  and  output.  There  are  a 
large  number  of  interconnections  between  portions  of  the 
database;  therefore,  altered  data  in  one  area  may  affect  the 
outcome  of  the  simulation.  Terrain  is  depicted  with  contour 
lines,  vegetation  and  cities.  Each  terrain  cell  has  a  fifty 
meter  resolution  which  corresponds  to  the  Defense  Mapping 
Agency  elevation,  vegetation  and  cultural  feature  description. 
Graphical  symbols  represent  one  or  more  systems  and  each 
system  can  have  one  or  more  weapon  [Ref  7]. 

A  Night  Vision  Electro-Optical  Laboratory  (NVEOL)  model  is 
used  for  detection.  Engagement  results  are  then  determined 
using  comparison  of  random  number  draws  to  a  probability  of 
hit  and  kill  database.   An  extensive  postprocessing  procedure 
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allows   for   collection   of   data   such   as   detection   and 
engagements  [Ref  8]. 

B.  TEST  DATA  CONVERSION 

The  PLS  files  from  the  five  field  trials  were  converted 
into  appropriate  Janus  databases  to  replicate  the  field  trials 
vehicle  and  helicopter  movement.  A  FORTRAN  program  designed 
by  Captain  Al  East,  a  former  student  at  the  Naval  Postgraduate 
School,  was  revised  to  read  the  PLS  data  files  and  arrange  the 
data  into  a  National  Training  Center  (NTC)  format  [Ref  9]. 
Another  FORTRAN  program  (INITNTC)  written  by  Mr.  Al  Kellner, 
a  programmer  from  TRADOC  Analysis  Command  (TRAC)  White  Sands 
Missile  Range,  was  used  to  convert  the  NTC  format  into  Janus 
format.  This  conversion  process  creates  scenarios  that 
replicate  the  force  structure  and  vehicle  routes  of  the  field 
trials.  At  this  point  the  modeler  has  the  option  to  adjust 
some  input  parameters  in  order  to  allow  the  simulation  to 
represent  the  field  trial  with  greater  accuracy. 

C.  INPUT  PARAMETERS 

The  Janus  database  was  not  changed  for  weapon  and  system 
characteristics.  The  data  was  previously  entered  by  students 
at  the  Naval  Postgraduate  School  in  conjunction  with  thesis 
work  and  is  assumed  to  be  accurate.  Since  the  altitude  of 
the  helicopters  in  the  field  trials  was  not  accurately  stated 
in  the  Test  Report  [Ref  4],  and  the  analyst  was  not  present, 
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there  is  uncertainty  in  the  flight  altitudes.  To  test  the 
sensitivity  of  altitude  change,  the  helicopters  altitude  was 
changed  and  each  trial  was  run  with  three  different  altitude 
levels:  15,  25,  and  35  meters.  Therefore,  the  five  Janus 
trials  became  fifteen  Janus  trials.  These  altitudes  were 
chosen  based  on  the  analysts  experience  as  a  pilot. 

D.  DATA  COLLECTION 

The  fifteen  Janus  trials  were  each  run  three  times.  This 
was  done  to  provide  enough  engagement  range  data  for  analysis. 
Each  scenario  was  run  interactively  at  first  to  insure  that 
the  simulation  and  the  entities  were  behaving  properly.  The 
three  runs  provided  a  total  of  10  to  50  helicopter  engagement 
ranges  for  each  trial.  The  Janus  postprocessor  was  used  after 
each  trial  to  provide  a  cumulative  list  of  engagements.  This 
list  was  manually  reviewed  to  provide  only  helicopter  first 
round  engagements. 

E.  DATA  LIMITATIONS 

The  PLS  files  did  not  account  for  the  repeated 
presentations  of  helicopters  in  the  field  trial.  This  created 
a  problem  with  the  converted  data.  If  a  helicopter  made  more 
than  one  presentation,  the  PLS  file  only  showed  one  long 
continuous  route.  This  made  it  necessary  to  manually  generate 
the  aircraft  presentations  and  routes  in  Janus.  Since  ninety 
percent  of  the  helicopter  routes  were  entered  manually,  there 
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may  be  errors  in  their  location.  The  Janus  screen  provides 
ten  digit  grid  coordinates,  while  the  human  can  not  really 
distinguish  to  this  accuracy.  This  can  cause  some  error  when 
engagement  ranges  are  determined  by  Janus.  If  the  location  of 
the  aircraft  is  incorrect  then  the  engagement  range  will  be  in 
error. 
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IV.   ANALYSIS 

A.  GENERAL 

The  purpose  of  this  comparison  is  to  determine  if  the 
means  and  distributions  of  engagements  are  the  same  for  the 
field  test  and  Janus.  If  they  are  different  then  a  possible 
trend  which  could  indicate  a  shift  in  location  or  a  possible 
error  might  be  of  interest.  Samples  from  the  field  test 
consisted  of  first  range  engagements  from  the  five  trials. 
Samples  from  Janus  consisted  of  three  runs  of  each  trial  for 
each  helicopter  altitude  15,  25 ,  and  35  meters.  There  was  a 
total  of  45  runs.  The  statistical  software  Statgraphics  [Ref 
10]  was  used  to  conduct  the  analysis. 

B.  ALTITUDE  SENSITIVITY  ^ 

Since  the  actual  altitudes  flown  by  the  helicopters  were 
not  known,  it  was  of  interest  to  see  if  the  altitude 
adjustment  used  in  Janus  would  provide  sufficient  differences 
to  affect  the  analysis.  This  sensitivity  analysis  could  help 
to  determine  the  best  set  of  data  for  comparison  with  the 
field  trials.  Each  scenario  was  run  with  the  three  different 
helicopter  altitudes  and  the  results  were  compared.  Three 
assumptions  were  made  concerning  the  Janus  data  for  this 
analysis.  The  data  was  assumed  to  be  normally  distributed, 
each  sample  is  a  random  sample  from  its  respective  population, 
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and  data  in  each  sample  are  independent  of  data  from  the  other 
samples.  The  medians  were  then  compared  using  a  notched  box 
and  whisker  plot  (Figure  2)  and  then  a  One  Way  Analysis  of 
Variance  (ANOVA)  was  done  to  test  the  null  hypothesis  that  the 
means  were  equal  (Table  2). 
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Figure    2    Notched   Box   and  Whisker   Plot 
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TABLE  2  ANOVA  VALUES  FOR  JANUS  TRIALS 


ANOVA 

TRIAL 

F-RATIO 

P -VALUE 

100 

2.736 

*.0700 

112 

.501 

*.6070 

122 

.351 

*.7060 

123 

.044 

*.9566 

125 

.439 

*.6461 

The  notched  box  plot  provided  an  approximate  5%  test  of  the 
null  hypothesis  that  the  true  medians  are  equal.  Since  the 
notches  overlap,  we  fail  to  reject  the  null  hypothesis  [Ref 
11].  With  the  ANOVA  analysis  we  failed  to  reject  the  null 
hypothesis  with  a  0.05  significance  level.  This  provides  no 
support  for  sensitivity  to  a  change  in  altitude  (for  these 
three  altitudes)  [Ref  12].  For  this  reason,  the  central 
altitude  of  25  meters  was  chosen  for  use  in  the  continuation. 
It  had  the  most  data  points.  A  summary  of  statistics  for  the 
Janus  (25  meter)  data  and  the  field  data  are  in  Appendix  A. 


C.   ANALYSIS  OF  NORMALITY  [Ref  12] 

Both  the  field  data  and  Janus  data  were  compared  to  the 
normal  distribution  (Figures  3  and  4)  via  Quantile-Quantile 
plot. 
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Figure  3  Field  Data  VS  Normal  Probability  Plot 
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Figure  4  Janus  Data  VS  Normal  Probability  Plot 
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The  field  data  appeared  close  to  normal,  but  the  Janus  data 
showed  greater  deviation.  This  was  especially  true  at  the 
higher  range  end  of  the  Janus  data.  This  could  be  due  to  the 
fact  that  Janus  would  not  engage  helicopters  beyond  the  range 
capability  entered  in  the  database.  It  seems  unwise  to  treat 
the  data  as  normal  because  of  these  deviations  from  straight 
lines.  A  less  formal  view  of  the  data  appears  in  the  Notched 
Box  and  Whisker  Plot  (Figure  5). 
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Figure  5  Notched  Box  and  Whisker  Plot 
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This  shows  at  a  glance  that  only  two  of  the  trials  (100  and 
122)  appear  to  have  equal  medians  using  a  95%  confidence 
interval . 

D.   NONPARAMETRIC  ANALYSIS  [Ref  13] 

Nonparametric  analysis  was  conducted  since  the  field  and 
Janus  data  could  not  be  supported  by  the  normal  distribution. 
The  assumption  that  the  data  was  Independently  Identically 
Distributed  was  made  pertaining  to  the  following  tests.  Both 
the  Kruskal-Wallis  and  Mann  Whitney  test  were  used  to  test  the 
null  hypothesis  that  the  means  are  equal  (Table  3). 


TABLE  3  KW/MW  P -VALUES 


TRIAL 

KRUSKAL-WALLIS 

MANN  WHITNEY 

P -VALUE 

P -VALUE 

100 

*. 28606 

*. 29133 

112 

.00131 

.00136 

122 

*.  26604 

*. 27954 

123 

.00001 

.00001 

125 

.00927 

.00964 

The  table  again  indicates  that  trial  100  and  122  fail  to 
reject  the  null  hypothesis  with  a  0.05  significance  level. 
These  results  agree  with  the  results  found  using  the  t-test 
and  ANOVA. 
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The  two  distributions  were  then  compared  using  the 
Kolmogorov-Smirnov  Two-Sample  Test  (Table  4).  This  test  was 
used  to  test  the  null  hypothesis  that  the  distributions  were 
the  same. 

TABLE  4  KS  TWO-SAMPLE  TEST  VALUES 


TRIAL 

KS  TWO-SAMPLE  TEST 

Test  Statistic 

P-Value 

100 

.2777 

*.3752 

112 

.5412 

.0118 

122 

.4545 

*.1865 

123 

.7352 

.000009 

125 

.5483 

.01099 

We  fail  to  reject  the  null  hypothesis  with  a  0.05  significance 
level  for  only  two  trials  100  and  122.  This  provides 
additional  support  for  the  previous  results. 

E.   FURTHER  ANALYSIS 

The  above  analysis  gives  us  the  notion  that  some,  but  not 
all,  Janus  runs  are  comparable  to  the  field  trials.  Let  us 
search  for  a  possible  trend  that  may  indicate  the  reason. 

1 .   Trimmed  Sample 

Janus  trial  125  has  the  most  outliers  (Figure  5),  so 
this  trial  was  chosen  for  application  of  a  t-test  with  trimmed 
data.  It  is  possible  that  some  extreme  points  in  the  Janus 
data  were  influencing  the  mean  and  distribution  to  shift,  and 
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therefore  leading  to  the  rejection  of  the  null  hypothesis.  A 
10%  trimmed  sample  was  compared  to  the  field  data  with  the 
null  hypothesis  that  the  trimmed  mean  equals  the  field  data 
mean.  The  result  was  hand  calculated  and  a  p-value  of  .00234 
was  determined.  This  shows  that  even  if  the  Janus  data  was 
trimmed  the  result  remains  to  reject  the  null  hypothesis  with 
a  0.05  significance  level. 

2.   Quantile-Quantile/Regression  Analysis 

Another  attempt  to  compare  the  distributions  was  done 
using  a  quantile-quantile  plot  for  both  the  engagement  range 
and  time.  The  plot  was  computed  based  on  interpolated 
quantile  values  for  the  Janus  data  [Ref  10:p.  55].  This  gives 
us  the  ability  to  look  at  the  specific  difference  in  each 
distribution  and  for  possible  trend  behavior  (Figure  6  and  7). 
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The  solid  line  is  the  x=y  line.  If  the  distributions  were  the 
same,  the  points  would  fall  on  this  line.  If  the 
distributions  are  different  and  some  trend  is  determined, 
scaling  laws  can  be  developed  which  would  allow  Janus  data  to 
be  converted  to  determine  field  trial  outcomes. 

The  data  points  were  then  fitted  with  a  simple 
regression  line  with  some  extreme  points  disregarded  [Ref  14]. 
The  dotted  line  on  the  graph  represents  this  regression  line 
and  the  circled  points  are  the  ones  selected  to  be  disregarded 
when  formulating  the  regression  equation.  The  regression  data 
(Table  5)  was  computed  based  on: 


Qxi  "  Janus  Quantile  Engagement  Range  Data 
QX2  -  Janus  Quantile  Time  Data 


Qyi  -  Field  Quantile  Engagement  Range  Data 
Qy2  -  Field  Quantile  Time  Data 


QY1  =  ai  +  bl   QX1 
QY2  =  a2  +  b2  QX2 

TABLE  5  REGRESSION  DATA 


TRIAL 

a, 

b, 

a, 

b, 

100 

343.49 

.87595 

4.2951 

1.7012 

112 

-2331.2 

1.0036 

29.744 

.32037 

122 

-11003 

2.3684 

-17.774 

.96152 

123 

-23656 

3.7393 

2.791 

1.1883 

125 

-4418.1 

1.414 

-3.2635 

.95837 
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There  does  not  appear  to  be  any  trend  or  similarities  in  any 
of  the  quantile  plots.  Each  regression  equation  varies 
dramatically  indicating  a  difference  due  to  scenario  change. 
Some  seem  to  be  merely  a  shift  in  location,  while  others  have 
a  completely  different  distribution.  There  does  not  appear  to 
be  one  specific  transformation  that  could  convert  the  Janus 
data  to  the  results  seen  in  the  field.  In  other  words,  the 
scaling  laws  appear  to  be  scenario  dependent. 
3.   Predicted  Values  Comparison 

A  final  attempt  to  find  some  link  between  the  Janus 
output  and  the  field  trial  was  done  using  the  simple 
regression  equations  formulated  from  the  Q-Q  plot.  The  raw 
data  from  the  Janus  trials  was  used  to  find  predicted  values 
for  the  field  trial.  The  purpose  was  to  seek  a  correlation 
between  time  of  engagement  and  its  range.  Such  a  correlation 
might  help  connect  the  scenarios  and  the  scaling  laws.  The 
scatterplots  are  displayed  in  Figure  8.  There  appears  to  be 
no  pattern  in  these  graphs,  again  giving  us  no  specific  link 
from  Janus  to  the  field  environment. 
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Figure  8  Predicted  Field  Values  Using  Regression  Equations 
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V.   CONCLUSIONS /RECOMMENDATIONS 

A.   CONCLUSIONS 

At  this  time,  Janus  should  not  be  accredited  for  post-test 
modeling  of  helicopter  engagements  by  the  LOS-F-H.  In  three 
of  the  five  scenarios  analyzed,  significant  differences  exist 
between  Janus  and  the  field  trial  helicopter  engagement 
ranges.  Janus  mean  engagement  ranges  were  higher  in  all  five 
cases  (Appendix  A) .  This  indicates  that  Janus  had  the  Line  of 
Sight  (LOS)  and  was  engaging  targets  prior  to  the  actual  field 
trial.   Several  reasons  are  offered  for  interpretation. 

1.  The  terrain  database  used  in  Janus  has  a  fifty  meter 
resolution.  Because  of  this,  many  lone  trees,  small 
rolling  hills  and  other  obstacles  are  not  replicated. 
These  obstacles  may  make  some  difference  in  the  LOS  for 
the  field  trials. 

2 .  Most  of  the  helicopter  routes  were  entered  into  Janus 
manually.  A  slight  inaccuracy  in  the  route  data  could 
result  in  a  helicopter  being  exposed  in  the  Janus  scenario 
while  it  was  not  in  the  field  scenario. 

3.  The  Janus  database  was  assumed  to  be  correct.  If  some 
parameters  were  inaccurate  it  can  affect  the  performance 
of  the  system  by  either  increasing  or  decreasing  its 
ability. 

4 .  The  actual  helicopter  altitude  was  unknown  since  the 
analyst  was  not  present  at  the  field  trial  and  the  after 
action  report  only  indicated  a  maximum  altitude.  The 
analyst  used  her  experience  and  judgement  to  determine  the 
helicopter  altitude  used  in  Janus.  The  three  altitudes 
tested  showed  no  sensitivity,  but  if  a  lower  altitude  was 
analyzed,  this  may  not  be  the  case. 
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5.  The  Janus  simulation  will  always  fire  at  its  first 
opportunity.  This  may  not  be  the  case  in  the  field 
environment.  Personnel  in  the  operational  test  may  not 
have  always  fired  at  their  first  opportunity.  This  may  be 
due  to  several  reasons  which  can  be  analyzed  further,  but 
are  not  in  the  scope  of  this  thesis. 

The  data  collection  limitations  in  the  field  test  were  minor 

and  the  analyst  does  not  believe  that  they  affected  the 

results . 

B .   RECOMMEND AT I ONS 

In  the  future,  if  the  Model-Test-Model  concept  is  to  be 
integrated,  modelers  must  be  present  through  the  entire  test 
sequence.  This  will  ensure  appropriate  interaction  between 
the  field  test  and  model.  The  modeler  will  have  no  unanswered 
questions  about  aircraft  altitude  or  other  field  trial 
specifics  and  can  ensure  that  the  appropriate  data  is  being 
supplied  by  the  field  test.  This  will  also  test  the  models 
data  base  to  ensure  the  systems  are  being  represented 
accurately  in  the  model. 

More  analysis  should  be  done  on  the  Janus  terrain 
database.  Fifty  meter  'resolution  does  not  provide  enough 
accuracy  to  replicate  the  field  test.  Future  studies  and 
analysis  using  higher  terrain  resolution  is  suggested. 

Although  this  analysis  could  not  accredit  the  Janus  model 
for  modeling  of  helicopter  engagements  by  LOS-F-H,  the  analyst 
believes  that  with  the  appropriate  database  and  involvement 
the  results  would  improve. 


32 


APPENDIX   A 


FIELD 

TRIALS 

TRIAL 

100 

112 

122 

123 

125 

SIZE 

18 

11 

11 

17 

12 

AVERAGE 

5719.11 

4835.73 

7119 

5469.41 

6611.75 

MEDIAN 

6040 

4786 

7011 

4849 

6819 

STD  DEV 

1505.11 

1769.65 

1828.74 

1835.4 

1031.94 

MIN 

3106 

2301 

4119 

2858 

4828 

MAX 

7954 

7975 

10426 

9065 

7816 

RANGE 

4848 

5674 

6307 

6207 

2988 

JANUS 

TRIALS 

TRIAL 

100 

112 

122 

123 

125 

SIZE 

27 

43 

12 

34 

31 

AVERAGE 

6128.44 

6858.16 

7652.75 

7747.26 

7509.52 

MEDIAN 

6472 

7094 

7729 

7740.5 

8125 

STD  DEV 

1675.81 

1317.79 

684.677 

559.393 

1275.37 

MIN 

2848 

4133 

6818 

6382 

4069 

MAX 

7889 

8489 

8335 

8458 

8478 

RANGE 

5041 

4356 

1517 

2076 

4409 

33 


LIST  OF  REFERENCE 

1.  Title  10  United  States  Code,  Section  13,  "Director  of 
Operational  Test  and  Evaluation". 

2.  Policy,  Operational  Test  and  Evaluation,  "Policy  for  the 
Application  of  Modeling/Simulation  in  Support  of  OT&E", 
24  January  1989. 

3.  Hollis,  Walter  W.  ,  "Verification  and  Validation  (V&V)  and 
Accreditation  of  Models",  Department  of  the  Army,  Office 
of  the  Under  Secretary,  Washington,  D.C.,  October  1989. 

4.  TRADOC  Test  and  Experimentation  Command,  Experimentation 
Center,  Line  of  Siqht-Forward-Heavy  Initial  Operational 
Test  (LOS-F-H  IOT)  Test  Reports,  Fort  Ord,  CA,  September 
1990. 

5.  Carrillo,  J.,  Walker,  A.,  and  RAM,  Inc.,  "Model-Test- 
Model  for  FAADS  Operational  Tests  -  Presented  at  the  58th 
Symposium  of  MORS",  unpublished,  12-14  June  1990. 

6.  Bundy,  D. ,  and  Creen,  M.  ,  "Generic  Model-Test-Model  Using 
High  Resolution  Combat  Models",  TRADOC  Analysis  Command, 
Monterey,  CA,  unpublished,  13  September  1991. 

7.  Department  of  Defense,  "Janus (A)  Analyst  Workstation  User 
Guide,  Version  3.0",  US  ARMY  TRAC  -  White  Sands  Missile 
Range,  NM,  August  1992. 

8.  Department  of  the  Army,  Janus (T)  Documentation,  TRAC  - 
White  Sands  Missile  Range,  NM,  June  1986. 

9.  East,  Allen  C,  "Comparison  of  Tank  Engagement  Ranges 
From  An  Operational  Field  Test  to  the  Janus (A)  Combat 
Model",  Naval  Postgraduate  School,  Monterey,  CA, 
September  1991. 

10.  Statistical  Graphics  Corporation  "Statgraphics, 
Statistical  Graphics  Systems",  University  ed.,  STSC, 
INC.,  1988. 

11.  Chambers,  John  M.  ,  Cleveland,  William  S.,  Kleiner,  Beat, 
and  Tukey,  Paul  A.,  Graphical  Methods  For  Data  Analysis, 
Wadsworth  and  Brooks,  1983. 

12.  Koopmans,  L.  H.,  Introduction  to  Contemporary  Statistical 
Methods,  Dunbury  Press,  Boston,  MA,  1987. 


34 


13.  Gibbons,  Jean  D.,  Chakraborti,  Subhabrata,  Nonpar ame trie 
Statistical  Inference,  3d  ed. ,  Marcel  Dekker,  Inc.,  New 
York,  NY,  1992. 

14.  Weisberg,  Sanford,  Applied  Linear  Regression,  2d  ed.  , 
John  Wiley  &  Sons,  New  York,  NY,  1985. 


35 


INITIAL  DISTRIBUTION  LIST 


No.  Copies 


Defense  Technical  Information  Center  2 

Cameron  Station 

Alexandria,  Virginia  22304-6145 

Library,  Code  52  2 

Naval  Postgraduate  School 
Monterey,  California  93943-5002 

Maria  C.  Pate  4 

TRAC/MTRY 

P.O.  Box  8692 

Monterey,  California  93943-5000 

Professor  Harold  J.  Larson,  Code  OR/La  1 

Department  of  Operations  Research 
Naval  Postgraduate  School 
Monterey,  California  93943-5000 

Professor  Robert  R.  Read,  Code  OR/Re  1 

Department  of  Operations  Research 
Naval  Postgraduate  School 
Monterey,  California  93943-5000 

CPT  Eugene  P.  Paulo  1 

TRAC/MTRY 

P.O.  Box  8692 

Monterey,  California  93943-5000 

LTC  Michael  D.  Proctor  1 

TRAC/MTRY 

P.O.  Box  8692 

Monterey,  California  93943-5000 


36 


DUDLEY  KNOX  ^,    ,  .. 

NAVAL  POSTGRADUATE  SCrlOUL 

MONTEREY  CA   93943-5101 


GAYLORD  S 


